Outcomes Measurement & Evaluation in Population Health using the ACG System

Stephen Sutch ^a, James Barrett ^b, Paul Molyneux ^b

Introduction
Risk adjustment applied to outcomes measurement can help to ensure that reasonable account are made of health factors that can effect outcomes that are not directly related to the intervention, programme or policy being reviewed. Common measurement approaches such as Case-mix adjustment, Stratified Sampling, Segmentation and Stratification can be used in Population Health.

This study utilises a Propensity score matching (PSM) method, as a quasi-experimental method which mimics randomization and creates matched-pair controls.

Outcomes do not directly assess quality of performance. They only permit an inference about the quality of the process. The degree of confidence in that inference depends on the strength of the predetermined causal relationship between process and outcome.

Data Needs - Because the relationship between process and outcomes is a probability, it is necessary to collect an appropriately large number of cases before one can infer if care is better or worse or meets specified standards. Outcome measurement requires specification of the appropriate time window which is the time when outcome differences caused by degrees of quality in health care are most manifest.

Data considerations include the need for comprehensive data, both with respect to the population of interest, but also across different health and social care providers.

Other data considerations: Availability, Completeness, Accuracy, Susceptibility to manipulation, Information about delayed outcomes, Data collection timeline.

Methods
Principally we want to know the participants' outcome with and without treatment controlling for all other effects. We know that participants differ from non-participants, so the objective is to find a large group of individuals who match the participants in all relevant pre-treatment characteristics. This then allows any difference (if well selected) to be attributed to the intervention. With multiple characteristics to control for a propensity score approach can be used. Such a score is based on the probability of participation in the program given the pretreatment characteristics.

PSM consists of: 1. score each patient using data prior to enrolment; then 2. pairing treatment and control individuals based on the same or nearest score; 3. follow-up and measure outcomes e.g. 6-month, 12 month, 24 month and compare results.

Results
Further selected results will be provided.




Discussion and Recommendations

• Establish measures and data collection from the outset, not retrospectively.
• Decide on randomised study, or casemix adjusted population cohorts.
• Is there an obvious comparison population (Intervention v Control)
• Matched pairs create a population similar to those in managed care ("Intervention group")
• Creation of a risk score or probability, assigned pre-enrolment.
• Consider the time frame (time window), is it absolute (same months), or did individuals/groups join at different times.
• Follow up measurement at specific time periods.
• Compare outcome measures of different groups.
• Create strata of sub-groups to better understand impact.

References

1. Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31-72.
2. Coca-Perraillon, M. (2007). Local and global optimal propensity score matching. SAS Global Forum 2007: Statistics and Data Analysis, Paper 185-2007.
3. Kleinman, K. (2010). Examples of tasks replicated in SAS and R: Example 7.35: Propensity score matching.

^a Bloomberg School of Public Health, Johns Hopkins University, United Kingdom
^b Johns Hopkins HealthCare Solutions, United Kingdom

Original Version in PDF